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Risk  Factors  versus  Dollar  Value:  Changing  How 
Weapon  System  Programs  are  Managed 


Robert  Murphy — Robert  Murphy  retired  as  a  member  of  the  Senior  Executive  Service  after  a 
33-year  career  in  the  US  Navy’s  Nuclear  Propulsion  Program.  He  finished  his  career  as  the 
Program’s  Director  of  Resource  Management,  responsible  for  budget,  acquisition  and  logistic 
support.  While  on  active  Government  service,  he  participated  in  the  acquisition  of  every 
nuclear  submarine  in  the  US  fleet  today,  in  addition  to  design  contracts  for  SEAWOLF  and 
VIRGINIA  Class  submarines.  Since  retiring  from  public  service,  he  has  been  consulting  for 
both  commercial  and  government  organizations,  specializing  in  major  system  acquisitions. 
Such  projects  have  included  several  cost  studies  of  VIRGINIA  Class  submarines,  in  addition 
to  advising  on  the  most  recent  multi-year  contract  for  acquisition  of  these  submarines.  RAND 
research  projects  have  included  several  studies  of  the  United  Kingdom  nuclear  submarine 
industrial  base.  Murphy  earned  an  MBA  from  George  Washington  University. 


Abstract 

The  author  discusses  the  current  basis  of  the  DoD’s  management  and  oversight  of 
MDAPs  (i.e. ,  their  dollar  value)  and  proposes  a  new  paradigm  in  which  the  level  of 
management  and  oversight  would  be  based  on  the  level  of  risk  an  MDAP  represents.  The 
author  also  examines  the  extent  to  which  the  DoD  is  prepared  to  assess  the  following 
categories  of  risk:  technical,  system  integration,  design,  production,  and  business.  Finally, 
the  author  makes  recommendations  to  improve  the  DoD’s  ability  to  assess  these  risks. 

Preface 

Today’s  defense  environment  is  placing  growing  pressure  on  defense  policymakers 
to  be  nimble  and  adaptive,  particularly  with  respect  to  acquisition  systems  and  processes. 
This  occasional  paper  is  one  in  a  series  drawing  upon  the  expertise  of  core  RAND 
Corporation  staff  to  explore  issues  and  offer  suggestions  on  topics  that  are  likely  to  be  of 
critical  importance  to  the  new  leadership:  the  use  of  competition,  development  of  novel 
systems,  prototyping,  risk  management,  organizational  and  management  issues,  and  the 
acquisition  workforce.  The  papers  are  designed  to  inform  new  initiatives  for  markedly 
improving  the  cost,  timeliness,  and  innovativeness  of  weapon  systems  that  the  Department 
of  Defense  (DoD)  intends  to  acquire. 

The  Office  of  the  Secretary  of  Defense  (OSD)  requires  review  of  Major  Defense 
Acquisition  Programs  and  decisions  by  senior  officials  on  the  basis  of  a  program’s  dollar 
value,  irrespective  of  risk.  In  this  paper,  we  propose  a  new  paradigm  in  which  the  level  of 
management  and  oversight  would  be  based  on  the  level  of  risk  a  program  represents, 
including  technical,  system  integration,  design,  production,  and  business  innovation  risk.  We 
also  examine  the  extent  to  which  the  DoD  is  prepared  to  assess  these  categories  of  risk  and 
identify  descriptive  levels  that  could  be  used  to  assess  and  categorize  design  and  business 
process  risk. 

This  study  was  sponsored  by  the  Office  of  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics  (OUSD(AT&L))  and  conducted  within  the  Acquisition 
and  Technology  Policy  Center  of  the  RAND  National  Defense  Research  Institute,  a  federally 
funded  research  and  development  center  sponsored  by  the  Office  of  the  Secretary  of 
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Defense,  the  Joint  Staff,  the  Unified  Combatant  Commands,  the  Navy,  the  Marine  Corps, 
the  defense  agencies,  and  the  defense  Intelligence  Community. 

For  more  information  on  RAND’s  Acquisition  and  Technology  Policy  Center,  contact 
the  Director,  Philip  Anton.  He  can  be  reached  by  e-mail  at  atpc-director@rand.org;  by  phone 
at  310-393-0411,  extension  7798;  or  by  mail  at  the  RAND  Corporation,  1776  Main  Street, 
Santa  Monica,  CA  90407-2138.  More  information  about  RAND  is  available  at  www.rand.org. 

Introduction 

Currently,  acquisition  programs  are  grouped  and  then  managed  at  the  Office  of  the 
Secretary  of  Defense  (OSD)  by  dollar  value —  depending  on  the  dollar  value,  the  OSD 
provides  different  levels  of  oversight  and  different  management  processes.  This  approach 
has  been  constantly  refined  over  the  years,  without  having  produced  any  noticeable 
improvement  in  terms  of  reducing  the  cost  growth,  schedule  slippage,  and  performance 
shortfalls  that  continue  to  plague  the  acquisition  of  weapon  system  programs.  This  paper 
argues  for  a  different  paradigm:  The  level  of  overall  risk  inherent  in  a  program  should  be  the 
main  basis  for  determining  the  process  and  level  of  review  a  project  should  receive.1 

Drawing  upon  examples  from  warship  acquisition  programs,  this  paper  also  argues 
that  inadequate  assessment  and  management  of  various  discrete  program  risks  results  in 
adverse  cost,  schedule,  and  performance  outcomes.  We  examine  existing  scales  for 
assessing  some  of  these  discrete  program  risks  and  make  recommendations  to  better 
assess  and  manage  several  programs  within  the  Defense  Acquisition  Management  System. 

Managing  by  Risk  Level  versus  Dollar  Value 

Currently,  the  OSD  requires  review  of  acquisition  programs  and  decisions  by  senior 
officials  on  the  basis  of  a  program’s  dollar  value,  irrespective  of  risk,  as  shown  in  Table  1 . 

However,  some  very  costly  projects  might  have  significantly  less  risk  than  projects  of 
similar  cost,  and  thus  should  require  less  oversight  as  well  as  the  use  of  different  criteria  at 
milestone  reviews.2  Conversely,  projects  may  cost  little  but  have  a  lot  of  risk  because  they 
tend  to  push  the  state  of  the  art  in  technology  and  may  also  involve  novel  business  or 
design  processes  that  may  require  more  comprehensive  oversight  than  just  dollar  value 
would  otherwise  indicate.  An  excellent  example  of  this  type  of  program — the  Advanced 
SEAL  Delivery  System  (ASDS) — was  discussed  in  a  May  2007  report  by  the  US 
Government  Accountability  Office  (GAO).  The  ASDS  is  a  Special  Operations  Forces’ 
battery-powered  submersible,  carried  to  a  deployment  area  by  a  submarine.  The  operating 
parameters  for  the  submersible  required  development  of  batteries  that  would  push  the  state 
of  the  art  in  that  technology.  The  initial  design,  construct,  and  deliver  contract  was  awarded 


1  Cost  is  a  factor  that  must  be  considered  when  determining  the  level  of  review.  A  multibillion  dollar 
program  requires  high-level  review  because  even  a  small  amount  of  cost  growth  involves  large  dollar 
amounts. 

2  For  example,  the  Navy  is  about  to  restart  construction  of  two  DDG  51-class  destroyers  at  a  cost  in 
excess  of  several  billion  dollars.  Over  60  destroyers  of  this  class  have  already  been  delivered  or  are 
in  the  final  stages  of  construction.  Because  of  this  track  record,  restarting  construction  of  two  new 
DDG  51s  will  no  doubt  expose  the  Navy  to  a  far  less  risk  of  adverse  cost,  schedule,  and  performance 
outcomes  than  construction  of  three  multibillion  DDG  1000-class  ships,  which  are  now  being 
designed  and  just  entering  construction. 
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for  $70  million  in  1994  for  delivery  in  1997;  because  of  the  dollar  value,  Milestone  Decision 
Authority  (MDA)  resided  with  the  Navy,  which  ultimately  accepted  delivery  of  ASDS  in  2003 
in  “as  is”  condition  at  a  cost  in  excess  of  $340  million.  The  GAO  concluded  that  “Had  the 
original  business  case  for  ASDS  been  properly  assessed  as  an  under-resourced,  concurrent 
technology,  design,  and  construction  effort  led  by  an  inexperienced  contractor,  DoD  might 
have  adopted  an  alternative  solution  or  strategy”  (GAO,  2007,  p.  13). 


Table  1.  Basis  and  Level  of  Program  Oversight 

(USD(AT&L),  2008) 


Program  Acquisition 
Category  (ACAT) 

Basis  for  ACAT 

Designation  Milestone 
Decision  Authority  (MDA) 

Milestone  Decision 

Authority  (MDA) 

1 

Estimated  total  expenditure 
for  research,  development, 
test,  and  evaluation 
(RDT&E)  of  more  than  $365 
million  or  for  procurement  of 
more  than  $2,190  billion 

ACAT  ID:  Under  Secretary  of 
Defense  for  Acquisitions, 
Technology,  and  Logistics 

ACAT  1C:  Head  of  DoD 
Component  (e.g.,  Secretary 
of  the  Navy)  or,  if  delegated, 
DoD  component  acquisition 
executive  (e.g.,  Assistant 
Secretary  of  the  Navy  for 
Research,  Development,  and 
Acquisition) 

II 

Estimated  total  expenditure 
for  RDT&E  of  more  than 
$140  million  or  for 
procurement  of  more  than 
$660  million 

DoD  component  acquisition 
executive  or  designate  (e.g., 
program  executive  officer) 

III 

Does  not  meet  criteria  for 
ACAT  II  or  above;  less  than 
an  MAIS  program  ACAT  ID: 
Under  Secretary  of  Defense 
for  Acquisition,  Technology, 
and  Logistics 

Designated  by  DoD 
component  acquisition 
executive  at  the  lowest  level 
appropriate  (e.g.,  program 
manager) 

NOTE:  Estimated  expenditures  are  in  FY  2000  constant  dollars. 


Focusing  on  Causes  Rather  than  Consequences 

Risk,  or  the  exposure  to  the  chance  of  failure,  is  a  word  heard  frequently  in  the 
acquisition  community.  All  acquisition  programs  face  risk  of  some  form  or  another.  Arguably, 
any  new  major  weapon  system  that  could  be  developed,  produced,  and  fielded  with  no  risk 
involved  is  probably  not  worth  acquiring. 

Overtly  or  otherwise,  much  of  a  program  manager’s  time  is  spent  managing  risk. 
After  all,  the  Defense  Acquisition  Management  System,  shown  in  Figure  1 ,  is,  in  essence,  a 
risk-management  process  designed  to  ensure  success  in  the  timely  delivery  of  weapon 
systems  that  meet  warfighter  requirements  while  staying  within  budget. 
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User  Needs 


Technology  Opportunities  &  Resources 


•  The  Materiel  Development  Decision  precedes 
entry  into  any  phase  of  the  acquisition 
management  system 

•  Entrance  criteria  met  before  entering  phase 

•  Evolutionary  Acquisition  or  Single  Step  to 
Full  Capability 
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Figure  1.  The  Defense  Acquisition  Management  System 

(USD(AT&L),  2008) 


The  risks  most  frequently  mentioned  by  defense  acquisition  officials  are  cost  growth, 
schedule  slippage,  and  performance  shortfalls.  This  is  not  surprising  as  cost,  schedule,  and 
performance  are  the  primary  attributes  by  which  programs  are  assessed  for  success  or 
failure.  Moreover,  the  Defense  Acquisition  University  (n.d.,  p.  2)  teaches  that  cost,  or 
performance  schedule,  and  performance  are  the  risk  factors  that  program  managers  must 
assess  and  manage  “to  ensure  that  DoD  is  acquiring  optimum  systems  that  meet  all 
capability  needs.” 


Assessment  of  cost,  schedule,  and  performance  is  clearly  a  management  task,  and  a 
good  program  manager  assesses  these  risks  using  periodic  data  accumulated  into 
management  reports  to  identify  problems,  regain  lost  ground,  and  then  stay  on  track. 
However,  these  are  broad  measures  of  risk.  A  better  program  manager  proactively  manages 
by  using  discrete  program  risks  and  submeasures  that  allow  him  or  her  to  look  ahead  and 
act  to  avoid  adverse  cost,  schedule,  and/or  performance  trends  and  outcomes.  In  other 
words,  managing  by  cost,  schedule,  and  performance  measures  is  akin  to  driving  a  car 
while  looking  solely  in  the  rearview  mirror — it  is  possible,  but  only  if  the  road  stays  straight.  A 
better  driver  looks  mostly  out  the  windshield,  with  only  an  occasional  look  in  the  mirror;  this 
driver  anticipates  and  easily  handles  curves  in  the  road. 


In  this  paper,  we  focus  on  five  discrete  programmatic  risk  categories: 


■  technical, 

■  system  integration, 

■  design, 

■  production,  and 

■  business. 


Taken  together,  these  risk  categories  portray  overall  acquisition  program  risk.3  They 
interact  in  numerous  ways  to  affect  a  project’s  cost,  schedule,  and/or  performance 
outcomes:  Obviously,  technologies  that  do  not  work  affect  performance,  but  so  can  poor 
business  decisions  that  increase  cost  and  lead  to  features  being  deleted  from  the  weapon 
system  to  remain  within  budget. 


3  For  simplicity,  risks  involved  in  fielding,  operating,  and  maintaining  the  weapon  system  are  not 
addressed  in  this  paper. 
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The  Defense  Acquisition  Management  System  appears  to  adequately  recognize  that 
incorporation  of  new  technologies  into  a  weapon  system  presents  risk,  providing  metrics  to 
systematically  assess  this  type  of  risk.  Time  is  also  provided  in  the  acquisition  process  for 
system  integration  matters  to  be  identified  and  resolved,  although  there  is  room  for 
improvement.  However,  as  will  be  discussed  in  subsequent  examples,  new  approaches  in 
design,  production,  and  business  areas  of  acquisition  programs  do  not  appear  to  receive  the 
same  skepticism  and  comprehensive  oversight  that  new  technologies  and  systems  receive. 

Well-Defined  Process  for  Assessing  Technical  Risk  Is  in  Place 

“Technical  risk”  is  exposure  to  the  chance  that  development  of  critical  technologies 
will  not  meet  program  objectives  within  cost  and  schedule  constraints.  In  assessing 
technical  risk,  program  managers  must  address  the  uncertainty  in  their  estimates  about  how 
much  time  and  effort  it  will  take  to  make  new  technologies  work.  The  importance  of  technical 
risk  is  well  understood  in  the  acquisition  community.  For  example,  DoD  guidance  states  that 
“the  management  and  mitigation  of  technology  risk... is  a  crucial  part  of  overall  program 
management  and  is  especially  relevant  to  meeting  cost  and  schedule  goals”  (USD(AT&L), 
2008,  para.  3.7.2.2). 

Technical  risk  is  also  extensively  addressed  in  the  Defense  Acquisition  Management 
System.  The  system  recognizes  evolutionary  acquisition  as  the  preferred  DoD  strategy  for 
rapid  acquisition  of  mature  technology  for  the  user.  One  purpose  of  evolutionary  acquisition 
(i.e.,  delivering  capability  in  increments  through  spiral  or  incremental  development)  is  to 
provide  time  to  better  manage  technology  risk  and  avoid  adverse  cost  and  schedule 
outcomes  that  often  result  from  trying  to  achieve  difficult  requirements  in  one  step. 

The  DoD  has  also  established  a  well-defined  process  based  on  Technology 
Readiness  Levels  (TRLs)  to  categorize  technical  risk  and  help  ensure  that  key  decision¬ 
makers  understand  the  risk  of  incorporating  different  technologies  into  weapon  system 
acquisition  programs  (the  TRLs  are  described  in  Table  2).  Using  this  process,  program 
offices  conduct  a  technology  readiness  assessment  under  the  auspices  of  the  DoD 
Component  Science  and  Technology  (S&T)  Executive;  the  Deputy  Under  Secretary  of 
Defense  (S&T)  evaluates  the  technology  readiness  assessment  and  forwards  findings  to  the 
Overarching  Integrated  Product  Team  leader  and  Defense  Acquisition  Board. 

The  TRLs  are  a  good  proxy  measurement  for  technical  risk:  The  lower  the  readiness 
level,  the  more  development  needed  to  incorporate  the  technology  into  a  weapon  system; 
and  the  more  development  needed,  the  greater  the  risk.  Overall,  technology  risk  has  been 
handled  fairly  well  in  warship  acquisition  programs,  which  tend  not  to  push  the  state  of  the 
art  in  technology  as  far  as  do  weapons  and  sensors.  A  recent  example  is  the  USS  Virginia, 
which  incorporates  various  new  technologies4  and  was  still  delivered  within  four  months  of 
the  original  schedule  established  a  decade  earlier  (Casey,  2007). 


For  example,  a  nonpenetrating  photonics  mast  versus  a 
t  Wide  Aperture  Sonar  Arrays,  etc. 
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Table  2. 


Technology  Readiness  Levels 

(DUSD(S&T),  2005) 


Technology  Readiness  Levels 


1.  Basic  principles  observed  and  reported 

1 .  Technology  concept  and/or  application  formulated 

2.  Analytical  and  experimental  critical  function  and/or  characteristic  proof 
of  concept 

3.  Component  and/or  breadboard  validation  in  laboratory  environment 

4.  Component  and/or  breadboard  validation  in  relevant  environment 

5.  System/subsystem  model  or  prototype  demonstration  in  a  relevant 
environment 

6.  System  prototype  demonstration  in  an  operational  environment 

7.  Actual  system  completed  and  qualified  through  test  and 
demonstration 

8.  Actual  system  proven  through  successful  mission  operations 
NOTE:  See  Mankins  (1995). 

System  Integration  Risk  Is  Assessed,  But  at  Later  Stages 

The  acquisition  community  also  assesses  system  integration  risk,  but  it  lacks 
effective  tools  to  measure  and  categorize  this  risk  early  in  a  program’s  life  cycle.  “System 
integration  risk”  is  exposure  to  the  chance  that  new  and  existing  technologies  being 
employed  in  a  weapon  system  may  not  work  together  and/or  interact  with  operators  and 
maintainers  to  meet  program  objectives  within  cost  and  schedule  constraints.  System 
integration  can  be  an  issue  within  an  individual  acquisition  program  (e.g.,  when  subsystems 
fail  to  interact).  It  can  also  be  an  issue  between  acquisition  programs:  Many  programs 
develop  capabilities  that  are  a  component  of  a  larger  warfighting  capability;  individually,  the 
component  programs  might  appear  to  be  a  low  or  moderate  risk,  but  in  combination  with 
other  programs,  the  overall  risk  might  be  much  higher  due  to  coordination  and  integration 
issues.  A  classic  example  occurred  during  the  Grenada  invasion  when  Army  and  Navy 
communications  systems  did  not  interact  well  during  the  joint  operation. 

System  integration  risk  is  extensively  treated  after  Milestone  B,  during  the 
engineering  and  manufacturing  development  (EMD)  phase,  at  which  time  a  program  should 
demonstrate  system  integration,  interoperability,  safety,  and  utility  (USD(AT&L),  2008,  para. 
3. 7. 1.1).  While  appropriate  attention  is  given  to  system  integration  risk  during  this  phase, 
this  assessment  occurs  after  the  second  of  three  milestones  in  the  process,  when  programs 
have  typically  built  up  so  much  momentum  that  they  are  difficult  to  stop,  regardless  of 
performance  or  progress.  Early  consideration  of  system  integration  risk — at  Milestone  A — by 
senior  decision-makers  could  result  in  developing  and  funding  integration-risk  mitigation 
plans  that  could  considerably  improve  acquisition  outcomes. 
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Combat  systems  in  warships  provide  an  example  of  the  problems  that  arise  when 
decisions  are  made  without  adequate  consideration  of  system  integration  risk.5  For  example, 
early  decisions  on  systems  architecture  and  processing  approaches  made  without  adequate 
consideration  of  risk  led  to  cost,  schedule,  and  performance  problems  with  submarine 
combat  systems  for  the  SSN  6881,  SEAWOLF,  and  Australian  Collins-class  submarines. 
According  to  a  report  for  the  Parliament  of  Australia  discussing  the  Collins-class  submarine, 

Of  the  early  decisions  in  the  Collins  program,  the  one  which  was  to  have  the  most 
public  effect  was  that  concerning  the  nature  of  the  vessels’  Combat  Data  System 
(CDS).  It  has  been  the  subsequent  failure  of  this  system  to  meet  its  design 
requirements  that  has  left  the  submarines  with  a  severely  impaired  combat  capability. 

By  the  end  of  1982... [the  Royal  Australian  Navy  (RAN)]  had  decided  that  the 
electronic  combat  systems  of  the  new  boats  would  be  fully  integrated.  Instead  of  the 
then  standard  central  computer  performing  all  data  analysis,  the  new  submarine  CDS 
would  use  a  data  bus  to  distribute  information  to  a  number  of  smaller  computer  work 
stations.  (Woolner,  2001) 

The  report  then  goes  on  to  discuss  the  lack  of  appreciation  for  the  risk  of  switching  to 
the  new  integrated  architecture  for  combat  systems. 

The  RAN  was  not  alone  in  its  “grand  folly.”...  The  Australian  information  technology 
(IT)  industry  assured  the  RAN  of  both  the  feasibility  and  inherent  advantages  of  a 
fully  integrated  combat  system  and  of  its  ability  to  contribute  to  such  a  program. 

Moreover,  the  RAN  was  not  the  only  navy  to  think  that  the  future  of  combat  data 
processing  lay  with  fully  integrated  systems.  The  USN  [U.S.  Navy]  specified  the 
same  concept  for  its  [BSY-2]  Integrated  Combat  System  for  the  U.S.  Navy’s  Seawolf 
class  nuclear  attack  submarines.  This  was  an  even  more  costly  failure  than  the 
Collins  CDS,  absorbing... $1.5  billion  [in  U.S.  dollars]  before  it  was  cancelled.6 

Tools  for  assessing  system-integration  maturity  earlier  on  have  been  proposed.  For 
example,  Sauser,  Ramirez-Marquez,  Magnaye,  and  Tan  (2008)  have  proposed  a  System 
Readiness  Level  (SRL)  index  that  would  incorporate  the  current  TRL  scale  as  well  as  an 
Integration  Readiness  Level  (IRL)  scale.  The  IRL  scale  they  describe  would  use  nine  levels, 
which  appear  compatible  with  the  widely  used  TRLs  and  appear  to  be  a  good  proxy 
measurement  of  system  integration  risk.  The  proposed  IRLs  are  listed  in  Table  3. 

The  Risks  of  Design  Process  Management  Are  Not  Well 
Understood 

“Design  risk”  is  exposure  to  the  chance  that  the  weapon  system’s  design  will  not 
result  in  effective  operation  or  be  easy  to  produce.  It  is  axiomatic  that  a  good  design  is 
essential  to  a  weapon  system’s  performance,  but  the  impact  of  design  on  a  weapon 
system’s  production  cost  and  schedule  outcome  is  not  as  well  appreciated.  However,  deci¬ 
sions  made  early  in  the  design  process  quickly  establish  not  only  the  performance  but  also 
the  ease  of  manufacture  and  resultant  cost  of  the  weapon  system.  While  the  ability  of  the 


5  A  combat  system  integrates  information  from  sensors,  synthesizes  this  information  for  combat 
commanders,  and  provides  fire  control  solutions  and  guidance  to  weapons. 

6  The  original  citation  mistakenly  attributed  this  to  the  BSY-1  program. 
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design  to  operate  effectively  can  be  considered  a  subset  of  technical  risk,  a  more  holistic 
approach  is  for  a  program  manager  to  assess  the  chance  that  the  design  process  to  be 
employed  for  the  weapon  system  will  generate  an  effective,  easy-to-produce  weapon. 

Table  3.  Integration  Readiness  Levels 

(Sauser  et  al. ,  2008) 


Integration  Readiness  Levels 


2.  An  interface  between  technologies  is  identified  with  sufficient  detail  to  allow 
characterization  of  the  relationship. 

1 .  There  is  some  level  of  specificity  to  characterize  the  interaction  (i.e., 
ability  to  influence)  between  technologies  through  their  interface. 

2.  There  is  compatibility  (i.e.,  a  common  language)  between 
technologies  to  orderly  and  efficiently  integrate  and  interact. 

3.  There  is  sufficient  detail  in  the  quality  and  assurance  of  the  integration 
between  technologies. 

4.  There  is  sufficient  control  between  technologies  necessary  to 
establish,  manage,  and  terminate  the  integration. 

5.  The  integrating  technologies  can  accept,  translate,  and  structure 
information  for  their  intended  application. 

6.  The  integration  of  technologies  is  verified  and  validated  with  sufficient 
detail  to  be  actionable. 

7.  Actual  integration  is  completed  and  mission  qualified  through  test  and 
demonstration  in  the  system  environment. 

8.  Integration  is  mission  proven  through  successful  mission  operations. 

The  design  process  necessary  for  an  effective  and  producible  weapon  system 
involves  complex  interactions  between  designers,  suppliers,  production  experts,  planners, 
and  estimators.  Design  process  complexity  has  also  increased  with  the  availability  of  more 
sophisticated  design  tools  such  as  electronic  product  models  and  computational  techniques 
(e.g.,  finite  element  analysis). 

Outcomes  from  two  current  acquisition  programs — the  United  Kingdom’s  ASTUTE- 
class  submarine  and  the  US  Navy’s  LPD  17-class  of  amphibious  transport  dock  ships — 
demonstrate  why  senior  decision-makers  in  the  OSD  acquisition  process  need  to  better 
understand  the  risks  new  design  processes  and  tools  present.  The  ASTUTE  was  the  first 
UK  submarine  to  be  designed  through  use  of  an  electronic,  three-dimensional  computer 
product  model.  The  prime  contractor’s  inability  to  manage  this  new  process  resulted  in 
extensive  delays  when  design  products  needed  to  build  the  ship  were  late.  General 
Dynamics  ultimately  had  to  be  hired  to  augment  and  manage  the  final  stages  of  the  sub¬ 
marine’s  detail  design  process.  Because  of  design  and  other  problems,  the  ASTUTE 
program  has  overrun  cost  greatly  and  is  years  behind  schedule. 

With  LPD  17,  the  US  Navy  competed  the  design  and  production  of  the  first  three 
ships  of  the  class  using  as  major  evaluation  and  award  criteria  (1)  the  plans  for 
accomplishing  detail  design  and  other  functions,  (2)  Integrated  Product  Data  Environment 
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(IPDE)  tools,  and  (3)  life-cycle  cost  projections;  these  criteria  were  given  more  weight  than 
price  (Comptroller  General  of  the  United  States,  1997).  The  then-Avondale  Shipyard  in  New 
Orleans,  Louisiana,  partnered  with  a  firm  that  was  developing  a  new  ship  design  IPDE  tool 
and  won  the  competition.  Subsequently,  the  LPD  17  experienced  considerable  cost  growth 
(about  70%)  and  schedule  delays  (CRS,  2008,  p.  12).  The  GAO  attributed  much  of  this  cost 
growth  to  the  new  design  tool: 

In  the  LPD  17  program,  the  Navy’s  reliance  on  an  immature  design  tool  led  to 
problems  that  affected  all  aspects  of  the  lead  ship’s  design.  Without  a  stable  design, 
work  was  often  delayed  from  early  in  the  building  cycle  to  later,  during  integration  of 
the  hull.  Shipbuilders  stated  that  doing  the  work  at  this  stage  could  cost  up  to  five 
times  the  original  cost.  The  lead  ship  in  the  LPD  class  was  delivered  to  the  warfighter 
incomplete  and  with  numerous  mechanical  failures.  (GAO,  2007) 

Senior  decision-makers  should  require  a  program  manager  proposing  to  use  new 
design  processes,  tools,  or  organizations  to  design  a  weapon  system  to  justify  selection  of 
the  new  process,  tool,  or  organization  and  develop  an  appropriate  risk  mitigation  plan.  An 
example  of  a  design  process  mitigation  plan  comes  from  the  VIRGINIA-class  submarine 
program.  Prior  to  VIRGINIA-class  construction  using  a  new  Integrated  Product  and  Process 
Development  (IPPD)  approach,  Electric  Boat  stated, 

a  representative  section  of  the  ship  about  a  year  early  with  a  portion  of  that  section 
started  about  two  years  early.  This  early,  controlled,  closely  monitored  ship 
construction  effort  ensured  thorough  preparation  for  full-ship  application  and  high 
confidence  in  the  new  process.  (General  Dynamics  Electric  Boat,  2002,  p.  33) 

Evaluation  of  Production  Risks  Lacks  Rigor 

An  earlier  and  more  rigorous  evaluation  of  production  risks  could  save  the  DoD  much 
difficulty  and  taxpayers  a  lot  of  money.  “Production  risk”  is  exposure  to  the  chance  that  the 
facility,  labor,  manufacturing  processes,  and  procedures  will  fail  to  produce  the  weapon 
system  within  time  and  cost  constraints.  Producibility — or  “production  capability” — is  a  func¬ 
tion  of  the  design;  production  facilities;  management  skills,  processes,  and  experience;  and 
workforce  skills  and  experience.  The  DoD  requires  assessment  of  contractors’  production 
capability  before  production  contract  award  in  the  production  and  deployment  phase,  but 
this  may  be  too  late  because,  at  this  point,  production  may  be  locked  in  by  the  organization 
that  won  the  design  contract.  Moreover,  in  the  authors’  experience,  and  as  exemplified  in 
the  LPD  17  source-selection  criteria  discussed  earlier,  the  production  category  of  risk  does 
not  receive  the  same  emphasis  in  selecting  a  shipbuilder  as  other  factors,  such  as  design 
concepts,  past  performance,  and  estimated  cost. 

The  Navy’s  DD  963-class  of  destroyers  and  LHA  1 -class  of  amphibious  assault  ships 
are  classic  examples  of  programs  in  which  the  DoD  considered  design  and  production  risk 
acceptable  when  awarding  contracts,  but  which  went  on  to  experience  about  the  worst  of 
every  production  factor  possible.  These  ships  presented  little  technical  and  system 
integration  risk,  but  ended  up  far  behind  schedule  and  over  cost,  due  in  part  to  identifiable 
production  risks.  Contracts  were  awarded  to  the  lowest  bidder,  Litton  Industries,  which 
owned  the  Ingalls  shipyard  in  Pascagoula,  Mississippi.  In  the  late  1960s,  Litton  Industries 
decided  to  invest  in  an  expansion  of  design  and  production  facilities  for  warships,  building  a 
new  shipyard  on  the  west  bank  of  the  Pascagoula  River,  across  from  its  existing  shipyard. 
The  new  shipyard  was  designed  to  be  operated  with  a  new  production  control  system  using 
modular  techniques  for  building  ships  (Northrup  Grumman,  2008). 
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After  the  award  of  the  LHA-  and  DD  963-class  contracts  to  Ingalls  for  nine  LHAs  and 
30  DD  963s  in  the  late  1960s,  Ingalls’  management  decided  to  shift  construction  of  some 
commercial  container  ships  from  the  old,  conventional  yard  to  the  new  facility  (Northrup 
Grumman,  2008).  The  expectation  was  that  doing  so  would  allow  the  new  facility  to  start  up 
and  have  any  problems  worked  out  while  the  LHA  and  DD  963  were  being  designed. 
However,  production  of  the  container  ships  using  the  new  control  system  led  to  delays; 
consequently,  the  ships  were  occupying  facilities  and  using  manpower  needed  to  start 
production  of  the  LHAs  and  DD  963s.  Production  of  the  LHAs  and  DD  963s  fell  far  behind 
and,  in  combination  with  other  problems  (design-related  issues,  inflation,  etc.),  the  costs 
were  overrun  substantially  and  the  ships  were  late  (GlobalSecurity.org,  2008). 

A  greater  emphasis  on  evaluating  production  risks  could  have  saved  an  enormous 
amount  of  time  and  money,  but  the  promised  cost  savings  resulting  from  construction  in  a 
new,  state-of-the-art  ship  fabrication  and  assembly  facility  proved  too  good  to  be  true.  The 
assessment  that  the  facility  would  be  derisked  by  building  container  ships  first  turned  out  to 
be  wrong,  and,  meanwhile,  two  entire  classes  of  ships  had  been  priced  and  placed  under 
contract. 

A  promising  approach,  initiated  by  the  Missile  Defense  Agency,  may  provide 
program  offices  across  the  DoD  with  better  insight  about  production  risk.  The  agency 
extended  the  notion  of  TRLs  to  engineering  and  manufacturing  by  developing  Engineering 
and  Manufacturing  Readiness  Levels  (EMRLs)  to  assess  the  maturity  of  a  program’s  design, 
related  materials,  tooling,  test  equipment,  manufacturing,  quality,  and  reliability  levels.  There 
are  five  EMRLs,  as  shown  in  Table  4. 

Table  4.  Engineering  and  Manufacturing  Readiness  Levels 

(DUSD(S&T),  2005) 


Engineering  and  Manufacturing  Readiness  Levels 


3.  System,  component,  or  item  validation  in  laboratory  environment  or  initial 
relevant  engineering  application  or  breadboard,  brass-board  development. 

1.  System  or  components  in  prototype  demonstration  beyond 
breadboard,  brass-board  development. 

2.  System,  component,  or  item  in  advanced  development.  Ready  for 
low-rate  initial  production. 

3.  Similar  system,  component,  or  item  previously  produced  or  in 
production.  System,  component,  or  item  in  low-rate  initial  production. 
Ready  for  full-rate  production. 

4.  Identical  system,  component,  or  item  previously  produced  or  in 
production.  System,  component,  or  item  in  full-rate  production. 

The  Risk  of  Early  Business  Decisions  Is  Not  Fully  Appreciated 

Business  decisions  made  early  in  a  program’s  life  can  significantly  affect  cost, 
schedule,  and  performance  outcomes.  “Business  risk”  is  exposure  to  the  chance  that  the 
overall  acquisition  strategy  for  a  program  will  not  result  in  the  desired  cost,  schedule,  and/or 
performance  outcomes.  Decisions  about  the  process  to  select  who  will  build  the  weapon 
system,  the  standards  to  which  it  will  be  built,  and  the  schedules  for  designing  and  building  it 
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all  entail  risk  that  is  not  always  appreciated  up  front.  To  evaluate  business  risk,  program 
managers  should  assess  the  following:  (1)  the  extent  to  which  the  acquisition  strategy  can 
result  in  selection  of  the  most  effective,  efficient  design  and  most  effective,  efficient 
production  entities;  (2)  whether  cost  estimates  and  schedules  are  valid;  (3)  whether  proper 
government  oversight  organizations  are  in  place;  and  (4)  whether  project  personnel  with 
proper  training  and  experience  are  available. 

A  good  example  of  early  business  decisions  gone  bad  is  the  Navy’s  Littoral  Combat 
Ship  (LCS)  Program.  The  lead  ship,  USS  Freedom  (LCS  1),  was  recently  delivered  after 
experiencing  substantial  cost  overruns  and  delivery  delays.  In  congressional  testimony 
given  to  explain  these  outcomes,  the  US  Navy  (2007)  identified  the  following  tenets  of  the 
new  business  model  used  to  acquire  the  LCS: 

■  Construction  of  LCS  seaframes  in  midtier  shipyards  that  “perform  predominately 
commercial  work,  maintaining  business  processes  and  overhead  structures  that 
keep  them  competitive  in  the  world  market”  (i.e.,  little  warship  experience).7 

■  “A  rapid  24-month  build  cycle  for  each  seaframe,  as  opposed  to  the  five  or  more 
years  that  have  become  the  norm  in  naval  shipbuilding.” 

■  “The  LM  lead  ship  detail  design  and  construction  effort  was  initiated 
simultaneously  and  the  lead  ship  commenced  construction  only  seven  months 
after  the  start  of  final  design  (i.e.,  concurrent  design/build).” 

■  “In  order  to  address  the  challenges  of  technical  authority  under  this  environment 
(reduction  in  NAVSEA  technical  personnel),  in  February  2003,  NAVSEA  and 
PEO  Ships  made  two  joint  decisions.  The  first  was  to  work  with  the  American 
Bureau  of  Shipping  (ABS)  to  develop  a  set  of  standards  (Naval  Vessel  Rules) 
that  could  be  applied  to  non-nuclear  naval  combatant  ships.  The  second  was  to 
utilize  ABS  to  class8  both  LCS  and  DDG  1000  using  the  new  rules.” 

No  doubt  there  were  good  arguments  for  these  individual  program  tenets.  However, 
the  cumulative  effect  of  the  risks  involved  in  building  a  new  design  warship  in  small 
commercial  shipyards  with  little  warship  experience  during  a  rapid,  concurrent  design/build 
process  and  to  a  set  of  technical  standards  themselves  under  development  appears  to  have 
been  greatly  underappreciated.  In  that  same  congressional  testimony,  the  Navy  identified 
cost  drivers  for  LCS  1  as  “concurrent  design-and-build  while  incorporating  Naval  Vessel 
Rules  (NVR),  reduction  gear  delays  created  by  a  manufacturing  error,  and  insufficient 
program  oversight”  (US  Navy,  2007).  The  risks  inherent  in  utilizing  an  entirely  new  business 
model  to  acquire  warships  were  obviously  neither  adequately  assessed  nor  managed. 

One  way  to  avoid  such  risk  would  be  to  require  program  managers  proposing  new 
and/or  radical  business  models  to  fully  justify  why  the  new  model  is  superior  to  past  practice, 
recommend  more  frequent  assessment  points  than  now  required  by  the  Defense  Acquisition 
Management  System,  and  incorporate  exit  strategies  in  contracts  for  the  government  to  use 
if  the  program  fails  to  meet  expectations. 


7  To  better  understand  the  differences  between  military  and  commercial  shipbuilding,  see  Birkler  et  al. 
(2005). 

8  The  American  Bureau  of  Ships  is  known  in  the  commercial  shipping  industry  as  a  “classification 
society,”  which  is  an  organization  that  sets  standards  for  design  and  construction  of  vessels  and 
integral  machinery  within. 
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Conclusions 


The  Defense  Acquisition  System  Framework  has  sufficient  tools  and  allows  time  for 
proper  assessment  and  management  of  technical  risk  and,  to  some  extent,  of  system 
integration  risk.  However,  design,  production,  and  business  risks  are  not  always  adequately 
assessed  and  managed.  As  shown  in  this  discussion,  scales  exist  that  represent  good  proxy 
measurements  of  technical,  systems  integration,  engineering,  and  production  risks;  what  is 
missing  are  descriptive  levels  that  could  be  used  to  assess  and  categorize  design  and 
business  process  risk.  We  recommend  that  the  DoD  explore  establishing  such  levels  and,  in 
Tables  5  and  6,  offer  starting  points  for  doing  so  (based  on  the  authors’  experience),  which 
may  help  program  managers  more  carefully  consider  these  risks. 

In  addition,  we  recommend  the  following  actions  to  better  assess  and  manage 
program  risk  overall: 

■  Assess,  categorize,  and  individually  review  each  technical,  system,  design, 
production,  and  business  risk  of  a  program  at  each  milestone  in  the  Defense 
Acquisition  Management  Framework. 

■  Require  program  managers  to  justify  new  or  radical  approaches  to  design, 
production,  or  business  processes  and  develop  and  implement  risk  mitigation 
plans  and/or  contract  off  -ramps. 

Table  5.  Proposed  Design  Process  Levels 


Design  Processes 


1.  New,  unproven  processes.  New  design  tools  under  development. 

New  design  organization. 

2.  Large  expansion  of  existing  design  organization.  Many  new  designers 
and  supervisors  unfamiliar  with  design  tools  and  processes. 

3.  Existing  design  organization  using  radically  changed  design  tools, 
processes,  and/or  technologies. 

4.  Experienced  design  organization  using  new  design  tools  with  proven 
processes. 

5.  Experienced  design  organization  using  existing,  proven  design  tools 
and  processes. 
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Table  6. 


Proposed  Business  Process  Levels 


Business  Processes 


4.  Using  a  new,  unproven  approach  to  source  selection.  Encouraging  new 

sources  of  supply.  Acquiring  new  technologies  without  well-established  cost¬ 
estimating  relationships.  Requiring  new  government  and/or  contractor 
organizations  to  be  formed. 

1 .  Using  new  procurement  process  in  established  industry.  Cost¬ 
estimating  relationships  only  at  high  levels.  Requires  expansion  of 
government  and  contractor  organizations. 

2.  Evolutionary  change  from  prior  acquisition  strategies.  Good  cost¬ 
estimating  relationships.  Existing  government  and  contractor 
organizations  can  easily  adapt  to  changes. 

3.  Using  same  approach  to  buying  similar  products.  Well-established 
cost-estimating  relationships  exist.  Experienced  government  and 
contractor  organizations  involved. 

4.  Acquiring  more  of  what  has  been  successfully  bought  before.  Using 
the  same  contractor  and  government  organizations. 

Although  such  tools  would  enhance  the  ability  of  program  offices  to  assess  and 
manage  risk,  the  DoD  should  also  consider  changes  in  oversight.  As  stated  at  the  outset  of 
this  paper,  the  current  acquisition  system  requires  review  and  decisions  by  senior  officials 
on  the  basis  of  a  program’s  dollar  value,  irrespective  of  risk.  A  better  use  of  their  limited  time 
may  be  to  focus  on  programs  with  high  risks,  letting  less-senior  officials  deal  with  lower-risk 
programs,  regardless  of  dollar  value.  For  example,  the  DoD  could 

■  lower  the  MDA  level  for  future  milestones  down 

— two  levels  for  programs  with  low  risk  in  all  risk  categories9 
— one  level  for  programs  with  moderate  risk  in  all  risk  categories.  10 

■  continue  to  follow  the  patterns  for  decision  authority  as  established  in  the 
Defense  Acquisition  Management  System  for  any  program  with  greater  than 
moderate  risk  in  any  of  the  five  categories  of  program  risk. 

In  this  way,  senior  decision-makers  might  be  able  to  better  concentrate  their  limited 
time  on  the  real  potential  problem  areas  in  a  program  before  problems  occur,  and  direct 
actions  to  be  taken  to  avoid  and/or  mitigate  potential  problems. 
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Current  risk  focus  in  DOD  acquisition 


Manage  by  dollar  value 

-  Program  acquisition  categories  stratified  by  $ 

-  The  higher  the  $,  the  higher  the  decision  authority 


-  Some  low  $  value  programs  could  benefit  from 
higher  level  scrutiny  (e.g.  ASDS) 

-  Some  high  $  value  programs  are  low  risk  (e.g. 
DDG  51  Class  restart) 


Causes  versus  consequences 

-  Are  cost,  schedule  and  performance  risks  that  can 
be  managed  or  consequences  of  discrete  risks? 
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Discrete  programmatic  risk  categories 


•  Technical 


-  Well  understood  in  acquisition  community 

-  Technology  Readiness  Levels 

•  Good  proxy  measure  for  technical  risk 

•  Well  defined  process  for  technology  readiness 
assessment 


-  Technical  risk  handled  well  in  ship  acquisition 
programs  (e.g.  VIRGINIA  Class) 
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Discrete  programmatic  risk  categories 


System  Integration 

-  Was  a  big  problem  with  submarine  combat 
systems  (e.g.  COLLINS,  SUBACS,  BSY  1  &  2) 

-  Extensively  treated  after  MS  B,  during  engineering 
and  manufacturing  development  (EMD)  phase 

-  Early  assessment  could  improve  outcomes 

-  Sauser  proposal  for  Integration  Readiness  Level 
scale 

•  Nine  levels,  compatible  with  TRLs 
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Discrete  programmatic  risk  categories 


•  Design 

-  Will  design  process  generate  an  effective,  easy-to- 
produce  weapon  system? 

-  Design  process  problems  can  lead  to  major  cost 
and  schedule  overruns  (e.g.  ASTUTE  &  LPD  17) 

-  Decision  makers  need  to  better  understand  the 
risks  new  design  processes  and  tools  present 


-  Program  managers  should: 


•  Justify  new  processes  and  tools 

•  Develop  appropriate  risk  mitigation  plan  (e.g.  VA  early 
ship  section  build  to  prove/debug  new  processes  &  tools) 
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Discrete  programmatic  risk  categories 


Production 

-  Production  readiness  review  before  production 
contract  award 

-  Earlier  program  decisions  may  lock  program  into 
production  contractor,  ready  or  not 

-  Expected  savings  from  new  production  facility  for 
LHA  1  and  DD  963  Classes  led  to  problematic 
contracts 

-  MDA’s  Engineering  &  Manufacturing  Readiness 
Levels  promising  tool  for  early  assessment 
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Discrete  programmatic  risk  categories 


Business 

-  LCS  represented  major  changes  to  Navy  ship 
acquisition  model 

•  Construction  in  commercially  competitive  yards 

•  Rapid  24  month  build  cycle 

•  Simultaneous  design  and  construction 

•  Use  of  ABS  Naval  Vessel  Rules 

-  Require  PMs  proposing  new  &/or  radical  business 
models  to: 

•  Justify  new  approach 

•  Incorporate  exit  strategies  in  Acquisition  Strategy 
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Conclusions  and  recommendations 


Better  &  earlier  attention  to  discrete  causes  of 
risk  can  lead  to  better  cost,  schedule  and 
performance  consequences 

Focus  oversight  on  risk  level,  not  $  value 

-  Assess  each  discrete  risk  at  each  milestone 

-  Require  PMs  to: 

•  Justify  new  or  radical  approaches  to  design,  production 
or  business  processes 

•  Develop  risk  mitigation  plans  &  exit  strategies 

-  Lower  the  Milestone  Decision  Authority  level  for 
oarams  assessed  with  low  risk 


